{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### kaggle insurance winner\n",
    "\n",
    "* **auto encoder** -- \n",
    "    *creating neural network features (embeddings, matrices) when we don't have dependent varibiable\n",
    "    * denoise auto-encoder to learn embeddings (15% randomly replace with some other row)\n",
    "    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IMDB\n",
    "\n",
    "* **vocablory** - list of all unique words that appear in all the documents.  \n",
    "* **bag of words** representation. i.e. of term document matrix. which words appear , how many times they appear. (no order). In **RNN**, can take use of  order of words also.  \n",
    "* **tokenization** - convert bag into list of words.  convert piece of text into list of tokens . every token is either word or single piece of punctuation.  \n",
    "    e.g. this \"movie\" isn't good will be tokenized to (this \" movie \" isn ' t good .) we don't want `good.` as an object. It means nothing.  \n",
    "* Fit tranform on training (to make vocublary from training) -> transform on test (apply in same order). but if test has some new words `unknown` category that we build in train.  \n",
    " \n",
    "* **Countvectorizer** : term document matrix  \n",
    "* **fit transform** term doc matrix form . stores sparse matrices in form (docID, wordID, no. of occur) . \n",
    "* **transform** \n",
    "* **vecrz.get_feature()** get feature names, words in this case . \n",
    "* **tokenization** splitting words in doc to make term doc matrix (seperate words in each row) . \n",
    "* **logistic regression** on val_term_doc handles when naive bayes does not follow independent assumptions. It handles linear relationship b/w variables (here variable are our words) . \n",
    "* **duel = True** in logistic regression. when you have more cols and rows\n",
    "* **ngrams** max_features - throws which ngrams do not exist much. like once etc. \n",
    "* **solver** -- (sag , \n",
    "    * conjugate gradient \n",
    "    * bfgs ("
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L.11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* weight decay -- e.g. L2 penaly = a*x^2. Derivate = weight decay = 2*a*x . \n",
    "* cross entropy for classification? --> \n",
    "* weights of 0 or 1 end up making it non opinionistic (posterior probability) . \n",
    "* **weight decay = 0.4** works pretty well in most of cases  \n",
    "\n",
    "* Sparse way of storing is to list out the indexes\n",
    "* one hot embedding -- mathematically identical to multiplying indexs matrix by one hot matrix. basically embedding means make a multiplication by one hot encoded matrix faster by replacing it with simple array lookup. \n",
    "\n",
    "* process - now have high dimentionality categorical variable -> convert to number from 0 to numb of levels --> learn a linear layer from that as if we had 1 hot encoded it without ever actually contructing the 1 hot encoded version and without doing matrix multiply -> just saving as index and doing array lookup -> in one hot encoded version everything was 0, no gradient. --> now gradient that are flowing back will update the particular row of the embedding matrix we have used ->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HW"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Understand gradients  \n",
    "* Look at `DotProdNB` function . \n",
    "* **Embedding matrix** see video . "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
